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Abstract. This study explores the potential of a feedback environment using simple 
string-based pattern matching technology for the provision of automated corrective 
feedback on cohesion problems. Thirty-eight high-frequent problems, including 
non-target like use of connectives and co-references were addressed providing both 
direct and indirect feedback. Advanced students of Gennan as a foreign language 
(L2) (n=36) received this feedback on summary writing in three subsequent sessions. 
Their revision activities were analysed for a ratio per 100 words and success rate, 
and their attitudes towards the feedback were investigated using questionnaires. The 
results show that automated feedback based on pattern matching has the potential to 
remedy over- and under-use of connectives and co-reference devices. Furthermore, 
although participants preferred direct feedback, the revision rate was higher with 
indirect metacognitive feedback providing grammar explanations. 

Keywords: automated feedback, advanced writing, cohesion, human computer 
interaction. 


1. Introduction 

Cohesion in written learner language is an important, yet under-researched area 
of investigation. L2 learners struggle with cohesion even at advanced levels of 
mastery. Cohesion can be expressed through a broad range of lexico-grammatical 
devices (Halliday & Hasan, 1976). Different lexico-grammatical preferences 
between the L2 and their mother tongue (LI) can cause L2 writers to make non¬ 
target like choices of cohesive features. Furthermore, the pedagogical approach 
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towards cohesion in L2 textbooks has been identified as harmful, as it is frequently 
characterised by a focus on grammatical accuracy, rather than on stylistic 
appropriateness (Cho & Shin, 2014). It is therefore timely to develop pedagogic 
support that addresses cohesion problems in learner writing in a more appropriate 
way. 

Automated feedback on L2 writing has long been applied mainly to address 
grammar, lexical, and/or mechanical issues in lower-level instruction. It is only 
recently that automated writing evaluation systems for English have adopted 
technologies that allow for a more sophisticated analysis of, and feedback on, 
learner writing with regard to cohesion (Crossley, Kyle, & McNamara, 2016). For 
German, no such tool has been developed yet. 

This explorative study investigates the potential of a tool using simple pattern 
matching technology for the provision of corrective feedback on cohesion 
problems. It also adds to the debate about the effectiveness of written corrective 
feedback in electronic writing environments (Bitchener, 2012; Ene & Upton, 
2014). It was driven by the following research questions: (1) Can cohesion 
problems successfully be addressed through automated feedback based on 
pattern-matching technology? (2) Does direct or indirect feedback trigger a 
higher amount of (successful) revisions? (3) What are students’ attitudes towards 
this feedback? 


2. Method 

2.1. Feedback creation 

The environment used for this study was developed at Ghent University 
(Belgium) for electronic teacher feedback. Among other features, this tool allows 
for the automated lookup of words and utterances in predefined lists, highlighting 
matches in students’ texts. By hovering over the highlighted words, additional 
information can be visualised. This feature was used in the study to provide 
automated feedback regarding potential non-target like use of cohesion devices. 
As cohesion is context bound by its very nature, the absence of context-sensitive 
lookup options and of Natural Language Processing (NLP) technologies for 
linguistic analysis posed a major challenge. Therefore, the feedback messages 
were formulated as non-directive suggestions inviting students to reflect their 
choice of cohesive devices. 
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To establish a list of potential cohesion problems in summary writing (the assignment 
used), a corpus of 72 summaries written by a comparable cohort of students to 
the participants in the study was scrutinised. Thirty-eight high-frequent problems 
emerged that can be identified through string-based pattern-matching. They mainly 
concern the non-target like use of connectives and co-reference devices. To target 
those 38 problems, 125 list entries were created covering morphological and lexical 
variations of the words or utterances involved. Three lists were produced, each 
conveying a different type of feedback: (1) direct feedback providing a suggestion 
for correction of the potential problem (DF), (2) indirect metacognitive feedback 
providing a grammar explanation (IF1), and (3) indirect feedback providing 
example utterances (IF2). 

2.2. Study design and data collected 

Thirty-six students of an intact writing class took part in the experiment. All 
participants have Dutch as their LI and major in German L2, with a medium 
advanced writing proficiency level 2 . Prior to the experiment, they received 
instruction on summary writing, with a focus on the use of cohesive devices. They 
also had one session to familiarise with the feedback tool. 

The participants received automated feedback on three summaries. On the first 
occasion, DF was provided, followed by indirect feedback in the two subsequent 
occasions. To avoid a sequencing effect, half of the group received first IF1 and 
then IF2 3 , and the other half vice-versa. Quantitative data of the revision activities 
were obtained through comparison of drafts before and after self-correction, and 
changes were coded according to successfulness. Furthermore, two questionnaires 
were administered to evaluate participants’ attitudes. 


3. Results and discussion 


3.1. Revision rate and success by feedback type 

Figure 1 displays the quantitative results of the revision analysis. To account for 
differences in word count in the three conditions, revisions per 100 words were 
calculated. IF1 triggered most revisions by far, totalling two and two and a half 
times the amount of DF and IF2, respectively. Furthermore, the success rate of 

2. B2-C1 on the common european framework of reference for languages scale 

3. See feedback examples: https://research-publishing.box.eom/s/4ebxytqvlcrnxml85ij5ilboc9ubv53q 
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revisions was higher following IF2 (74% of all changes induced by this feedback 
type) than following DF (64%) and IF1 (61%) 4 . These findings are not in line with 
the results of earlier studies on electronic (human) feedback on complex L2 writing 
that ascribed a higher success rate to direct, explicit feedback (c.f. Ene & Upton, 
2014, p. 89). 

Figure 1. Revision rate and success by feedback type 


Revisions /100 words 



Condition 1 (DF) Condition 2 (IF1) Condition 3 (IF2) 



□ successful ■ unsuccessful □ neutral changes 


3.2. Students’ attitudes towards the feedback 

Three results obtained from the questionnaires will be discussed here. They concern 
(1) preferences for feedback type, (2) overall attitudes towards usefulness of the 
automated feedback, and (3) attitudes with regard to the unnecessary feedback 
created due to lack of sophisticated technology. 

(1) Contrary to expectations based on the amount of revisions, students did not 
prefer indirect feedback based on grammar explanations, which actually received 
the lowest preference score (28%), followed by indirect feedback based on 


4. See revision examples: https://research-publishing.box.eom/s/4ebxytqvlernxml85ij5ilboc9ubv53q 
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examples (31%). Students’ comments comparing the two indirect feedback can 
be interpreted as windows to their preferred learning styles, i.e. inductive learning 
(“1 like to deduce rules from examples; grammar explanations are tedious”) versus 
deductive learning (“I prefer an explanation because I want to know why things 
are wrong”). The majority of students, however, preferred direct feedback (41%). 

(2) Overall, students were positive with regard to the feedback received, rating 
the usefulness for self-correction in the concrete assignment 3 (50%) or 4 (41%) 
on a Likert scale of 5 (1= not useful, 5 = very usefiil). Furthermore, 71% of the 
students declared they would like to use the tool in self-directed study mode to 
revise their bachelor papers. Regarding the usefulness of the feedback as a tool to 
raise awareness for potential problems in their writing in general, the satisfaction 
scores were even higher: 3 (24%), 4 (53%), and 5 (15%). Indeed, students declared 
to look at their written texts “with different eyes” after having received automated 
feedback on cohesion three times in a row. This suggests that the feedback was 
helpful to increase their noticing ability (Schmidt, 1990), which is important for the 
development of learner autonomy. 

(3) As mentioned above, one of the main challenges in the study design was the 
lack of context-sensitive lookups which lead to a high amount of unnecessary 
highlighting of words and utterances that actually were well formulated. As this 
might have a potentially overwhelming effect, it was important to elicit students’ 
attitudes about this aspect. Interestingly, only 6% of the students rated the 
unnecessary feedback annoying (4 on a Likert scale ranging from l=not annoying 
at all to 5=very annoying), while 15% felt not annoyed at all (1 on Likert scale). 
The majority rated their degree of annoyance as 3 (47%) or 2 (35%). 


4. Conclusions 

This explorative study provided evidence that cohesion problems can successfully 
be addressed through automated feedback. Simple pattern-matching technologies 
in combination with tentatively formulated suggestions can help students to 
remediate over- and under-use of connectives and co-reference devices. While 
students preferred direct feedback, indirect feedback proved to be more successful 
for revision, with grammar explanations triggering a higher amount of revisions, 
and example utterances leading to a higher rate of successful revisions. These 
findings add to our understanding about the effectiveness of written corrective 
feedback, adding non-directive automated feedback as a new, under-researched 
setting. Perhaps even more important than revision success, students reported that 
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the repeated provision of feedback through highlighting potentially problematic 
words and utterances increased their noticing. These promising results indicate 
that automated feedback on cohesion is a path worth further exploration. For future 
tool development, implementing NLP for linguistic and semantic analysis should 
be considered. This way, the range of cohesive devices to be addressed can be 
expanded and unnecessary feedback can be avoided. 
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